From Cluster to Cloud to Appliance

We propose a client-side virtual machine (VM) as an alternative to on-campus clusters and off-campus clouds as a development environment for students in introductory courses. In Fall 2011, we deployed the CS50 Appliance, our own such VM, to 600 students on campus and, in Fall 2012, to 700 students on campus and 140,000 students online. We present in this work the results of that two-year experiment. The appliance itself is available as open source for others to adapt or adopt. Not only did the appliance enable us to provide students with simpler tools, among them a graphical editor without any latency, it also enabled us to provide more sophisticated tools too, including a web server and database server. Moreover, the appliance ensured that the course's workload no longer required constant Internet access, particularly of students abroad. And the appliance alleviated load on the course's servers, with execution of students' programs now distributed across students' own CPUs. Without the appliance (or more costly clusters or clouds), we certainly could not have accommodated as many as 140,000 students. But some students' laptops, particularly netbooks, struggled under the appliance's weight. Even though designed to be lean, the appliance, like any VM, still consumes resources, particularly RAM. And unforeseen technical difficulties arose in both years, most, but not all, of which we redressed with mid-semester updates and documentation. Overall we have judged our deployment of an appliance a success, superior to past years' clusters and clouds. And we continue to refine the appliance for Fall 2013.Engineering and Applied Science

Virtualizing Office Hours in CS 50

In Fall 2007, we introduced “virtual office hours” into Harvard College’s introductory computer science course, CS 50, so that students could meet with teaching fellows (TFs) online to discuss problem sets at any hour from anywhere. Our goals were to lower the bar to interaction among TFs and students and to improve the efficiency and convenience of the same. Rather than rely on email and online forums alone, we experimented with Elluminate, third-party software that not only allowed students and TFs to chat via IM and VOIP, it also enabled the latter to see and even share control of the former’s screens (e.g., code in students’ terminal windows). Students, in turn, were able to troubleshoot bugs with TFs by their (virtual) side. We surveyed our nearly 300 students on their experiences with office hours, both physical and virtual. Although most students responded positively to the idea of virtual office hours, only 55% logged in at least once. However, nearly the same number (62%) attended the physical. We ultimately judged our virtual office hours a net positive, with 14% of students attending the virtual (and 21% the physical) “often.” But our experiment was not without some unexpected results. We found that wait times online sometimes matched or exceeded those in the physical lab, partly the result of the software’s own shortcomings and students’ habits online. Ultimately, the audience for these virtual office hours was entirely self-selecting. Those students who liked the experience online opted in, whereas those who preferred more traditional help opted out.Engineering and Applied Science

CS50 Sandbox: Secure Execution of Untrusted Code

We introduce CS50 Sandbox, an environment for secure execution of untrusted code. Implemented as an asynchronous HTTP server, CS50 Sandbox offers clients the ability to execute programs (both interactive and non-interactive) written in any compiled or interpreted language in a tightly controlled, resource-constrained environment. CS50 Sandbox’s HTTP-based API takes files, command lines, and standard input as inputs and returns standard output and error plus exit codes as outputs. Atop CS50 Sandbox, we have built CS50 Run, a web- based code editor that enables students to write code in a browser in any language, whether compiled or interpreted, that’s executed server-side within a sandboxed environment. And we have built CS50 Check, an autograding framework that supports black- and white-box testing of students’ code, leveraging CS50 Sandbox to run series of checks against students’ programs, no matter the language of implementation. We present in this work the pedagogical motivations for each of these tools, along with the underlying designs thereof. Each is available as open source.Engineering and Applied Science

Moving CS50 into the Cloud

In Fall 2008, we moved Harvard College’s introductory computer science course, CS50, into the cloud. Rather than continue to rely on our own instructional computing infrastructure on campus, we created a load-balanced cluster of virtual machines (VMs) for our 330 students within Amazon Elastic Compute Cloud (EC2). Our goals were both technical and pedagogical. As computer scientists, we wanted more control over our course’s infrastructure (e.g., root access), so that we ourselves could install software at will and respond to students’ needs at any hour without an IT department between us and our systems. As teachers, we wanted easier access to our students’ work (as via su) as well as the ability to grow and shrink our infrastructure as problem sets’ computational requirements demanded. But we also wanted to integrate into the course’s own syllabus discussion of scalability, virtualization, multi-core processing, and cloud computing itself. What better way to teach topics like those than to have students actually experience them. Although Amazon supported our experiment financially with credits, it was not without costs. Serving as our own system administers cost us time, as did some self-induced late-night technical difficulties. But the upsides proved worth it, as we accomplished our goals. We present in this paper what we did right, what we did wrong, and how we did both so that others can more easily build their own home in the cloud.Engineering and Applied Science

Reinventing CS50

Computer Science 50 is Harvard College’s introductory course for majors and non-majors alike, enrollment in which both rose and fell along with the dotcoms. Although enrollment peaked in 1996 at 386 students, it had settled by 2002 in the neighborhood of 100. We set out in 2007 to combat that trend by tackling two problems. We hypothesized that CS50 suffered from two, one of perception and one of design. Although, per end-of-term surveys, the course had never lacked for good teachers or good content, the consensus on campus for years had been to beware this particular course. And the course’s own syllabus may very well have been dated in the eyes of students who had begun to carry regularly modern hardware and software in their backpacks and pockets. Not only did we proceed to revamp every one of CS50’s problem sets, we brought its syllabus more in line with technological trends already familiar to students. And we altered the tone of the course to appeal to those “less comfortable” with computing on campus. But we took care to preserve the course’s rigor and underlying fundamentals, lest we do our own students a disservice. Our new approach appears to be working. Between 2006 and 2007, enrollment in CS50 more than doubled from 132 to 282 (+114%). Between 2007 and 2008, enrollment increased another 17% to 330, though even more striking was that year’s 48% increase in female enrollment. By 2009, enrollment remained strong at 338. We present in this work what we have done and why we have done it.Engineering and Applied Science

Streamlining Grading toward Better Feedback

CS50 is Harvard University's introductory course aimed at majors and non-majors alike. Each week, students complete programming assignments and have traditionally received feedback from staff in the form of comments on PDFs of their code. Staff have historically reported spending significant amounts of time grading because of bottlenecks that included generating PDF documents and manually emailing feedback to students. Because we preferred that staff spend less of their time on grading logistics and more time providing feedback and helping students online or in person, we set out to improve the efficiency of the grading process. In Fall 2012, we developed and deployed CS50 Submit, a web-based utility through which staff can leave feedback for students via inline "sticky notes." Following the introduction of CS50 Submit, staff reported grading for 10% fewer hours (i.e., 42 minutes) per week and 13% fewer minutes (i.e., 4 minutes) per student, even while providing as much or more feedback. Meanwhile, we observed significantly higher levels of engagement with the course's online discussion board among staff, suggesting a more favorable distribution of staff workload. With CS50 Submit, we have also been able to audit exactly how much time staff spent grading each week in order to identify additional bottlenecks. Using CS50 Submit, we also observed that, on average, 9% of students each week never read their graders' comments, with a peak one week of 14%. The number of students who did not read feedback increased with time, which has led us to question whether asynchronous, textual comments are the most effective feedback mechanisms for students. In future terms, we plan to experiment with in-person, interactive means of delivering feedback to students. In this paper, we present CS50 Submit and the insights it has yielded into the behavior of students and staff alike.Engineering and Applied Science

Scaling Office Hours: Managing Live Q&A in Large Courses

Computer Science 50 (CS50) is Harvard University’s introductory course for majors and nonmajors alike. So that students have an adequate support structure with which to tackle the course’s weekly programming assignments, we offer weekly “office hours,” during which students can receive one-on-one help from teaching assistants. In Fall 2010 and years prior, office hours were held in a basement-level computer lab. However, this environment did not appeal to staff or students. Moreover, this format for office hours suffered from logistical inefficiencies, repetition of questions among students, and lack of communication among staff, which led to high wait times for students. We relocated office hours in Fall 2011 to dining halls to create a more social and collaborative workspace, with more staff on duty at once. We also developed CS50 Queue, a web- and iPadbased system for managing office hours’ logistics. Overall, the new format proved a success. Attendance at office hours grew more than linearly, with an average of 120 students attending per night, up from 30 students in 2010 despite only a 23% increase in enrollment. Even though Queue enabled us to scale, new logistical challenges arose, and wait times for students still sometimes exceeded an hour. We intend to address those challenges in Fall 2012 in order to reduce wait times to 15 minutes at most.Engineering and Applied Science

Engaging Students through Video: Integrating Assessment and Instrumentation

CS50 is Harvard’s introductory course for majors and non-majors alike. For years, we have posted videos of the course’s lectures and sections online for the sake of review and distance education alike. But students’ experience with these videos has been historically passive. Students have been able to watch the course’s content on demand, rewinding and fast-forwarding at will, but they have not had means to engage interactively with the content or to check their understanding of material while watching videos. Furthermore, while we collected basic usage data (e.g., how many times a video was viewed), we lacked detailed analytics describing, for example, which portions of a video were commonly skipped or watched multiple times by students. To make videos more immersive and engaging for students, we developed CS50 Video, an open-source video player for desktop and mobile devices. CS50 Video allows instructors to integrate assessment questions to be answered by students at their own pace or at specific points in time directly into a video player. CS50 Video also allows students to search over video transcripts to find content easily as well as view videos at variable playback speeds (in order to make videos more accessible for ESL learners). Finally, CS50 Video integrates with third-party analytics solutions to allow instructors to view detailed usage statistics describing how students are interacting with videos (e.g., which videos or portions of videos are commonly watched or skipped over). We have deployed CS50 Video to students taking CS50 online and have obtained preliminary results. Because CS50 Video stores responses to questions server-side, we have been able to track students’ performance on in-video assessments. Thus far, we have observed that only 28% of students who watch online videos have engaged with assessment questions. Students who answer an assessment question incorrectly on their first attempt will often try again until reaching a correct answer, with 84.5% of correct answers reached in at most three attempts. We next plan to analyze the effects of in-video assessments on students’ mastery of material and introduce A/B-testing functionality for questions. We also plan to use students’ performance on assessments to understand the topics with which students struggle.Engineering and Applied Science

Hydrogen Epoch of Reionization Array (HERA)

The Hydrogen Epoch of Reionization Array (HERA) is a staged experiment to measure 21 cm emission from the primordial intergalactic medium (IGM) throughout cosmic reionization ($z=6-12$), and to explore earlier epochs of our Cosmic Dawn ($z\sim30$). During these epochs, early stars and black holes heated and ionized the IGM, introducing fluctuations in 21 cm emission. HERA is designed to characterize the evolution of the 21 cm power spectrum to constrain the timing and morphology of reionization, the properties of the first galaxies, the evolution of large-scale structure, and the early sources of heating. The full HERA instrument will be a 350-element interferometer in South Africa consisting of 14-m parabolic dishes observing from 50 to 250 MHz. Currently, 19 dishes have been deployed on site and the next 18 are under construction. HERA has been designated as an SKA Precursor instrument. In this paper, we summarize HERA's scientific context and provide forecasts for its key science results. After reviewing the current state of the art in foreground mitigation, we use the delay-spectrum technique to motivate high-level performance requirements for the HERA instrument. Next, we present the HERA instrument design, along with the subsystem specifications that ensure that HERA meets its performance requirements. Finally, we summarize the schedule and status of the project. We conclude by suggesting that, given the realities of foreground contamination, current-generation 21 cm instruments are approaching their sensitivity limits. HERA is designed to bring both the sensitivity and the precision to deliver its primary science on the basis of proven foreground filtering techniques, while developing new subtraction techniques to unlock new capabilities. The result will be a major step toward realizing the widely recognized scientific potential of 21 cm cosmology.Comment: 26 pages, 24 figures, 2 table